Support the Replit Audio messages#1525
Conversation
lhchavez
force-pushed
the
replit-audio-extension
branch
from
8d8da8d to
4bdcdbe
Compare
These messages are intended to allow clients to request the server to send compressed audio (with the Opus and MP3 codecs) in push and pull models. The discussion is in https://groups.google.com/g/novnc/c/pQ6h2xLf3Kc Reference implementation (client-side): novnc/noVNC#1525 Reference implementation (server-side): https://github.com/replit/rfbproxy Signed-off-by: Luis Héctor Chávez <luis@repl.it>
This change: a) Adds a button to toggle the audio stream. This is needed since most browsers require the audio play action to be run in a user event handler, so having the toggle audio be an explicit button achieves that (and is also to avoid annoying users). b) Drops support for the QEMU audio extension and instead uses the Repl.it Audio messages, which allows for explicit negotiation for audio compression. c) Makes the audio library more robust and now works in Chrome and Firefox.
lhchavez
force-pushed
the
replit-audio-extension
branch
from
4bdcdbe to
70c57e8
Compare
This change: a) Adds a button to toggle the audio stream. This is needed since most browsers require the audio play action to be run in a user event handler, so having the toggle audio be an explicit button achieves that (and is also to avoid annoying users). b) Drops support for the QEMU audio extension and instead uses the Repl.it Audio messages, which allows for explicit negotiation for audio compression. c) Makes the audio library more robust and now works in Chrome and Firefox. (cherry-pick of novnc#1525)
|
I'm trying to test this but I'm getting very choppy audio. Any suggestions on debugging and getting it working? |
oh no! does the rfbproxy log mention a lot of reconnects? if not, you can add this diff to diagnose it client-side: diff --git a/core/util/audio.js b/core/util/audio.js
index dbfc400..40b37c6 100644
--- a/core/util/audio.js
+++ b/core/util/audio.js
@@ -125,6 +125,12 @@ export default class AudioStream {
// seeked to the latest possible position that doesn't trigger buffering
// to avoid an arbitrarily large desync between video and audio.
_appendChunk(timestamp, chunk) {
+ console.debug("appending chunk", {
+ timestamp,
+ currentTime: this._audio.currentTime,
+ buffered: this._audio.buffered.length && this._audio.buffered.end(this._audio.buffered.length - 1),
+ });
+
this._audioBuffer.appendBuffer(chunk);
if (
timestamp - this._audio.currentTime > MAX_ALLOWABLE_DESYNC &&are both connections local, or are you using the PoC that i shared before? |
These messages are intended to allow clients to request the server to send compressed audio (with the Opus and MP3 codecs) in push model. The discussion is in https://groups.google.com/g/novnc/c/pQ6h2xLf3Kc Reference implementation (client-side): novnc/noVNC#1525 Reference implementation (server-side): https://github.com/replit/rfbproxy Signed-off-by: Luis Héctor Chávez <luis@repl.it>
|
Sorry about the delay. I could spend a little time on this today, but unfortunately not a lot. The proxy doesn't log anything so I'm afraid there is no help there. Without any modifications I'm getting this in the browser log: With that patch I'm getting a bunch of lines, but nothing obviously wrong: Whole log: console-export-2021-7-23_15-0-12.txt |
gotcha. let me build a Docker container so that we're working with the same environment, and have all the logging. |
|
@lhchavez, were you able to get a docker image up for testing? |
|
not yet, i've been on vacation for a couple of weeks. i'll give it a try tomorrow! |
got it: FROM ubuntu:groovy
RUN DEBIAN_FRONTEND=noninteractive apt-get update -y && \
DEBIAN_FRONTEND=noninteractive TZ=UTC apt-get install \
--no-install-recommends -y \
curl \
fluxbox \
gpg \
gpg-agent \
libmp3lame0 \
libopus0 \
libpulse0 \
mplayer \
pulseaudio \
software-properties-common \
tigervnc-standalone-server \
vlc \
xterm \
xz-utils && \
rm -rf /var/lib/apt/lists/*
RUN curl -sL https://github.com/replit/rfbproxy/releases/download/v0.1.1-b81eb2f/rfbproxy.tar.xz | tar xJ -C /
RUN useradd --create-home runner
WORKDIR /home/runner
USER runner
ENV DISPLAY=:0
# Verbosity can be upped by changing `RUST_LOG=debug` in the script below.
RUN echo $'\n\
#!/bin/bash\n\
Xtigervnc --SecurityTypes=None --rfbport=5901 --localhost "${DISPLAY}" 2>&1 | sed "s/^/[Xtigervnc ] /" & \n\
sleep 1 \n\
RUST_LOG=info rfbproxy --enable-audio 2>&1 | sed "s/^/[rfbproxy ] /" & \n\
sleep 1 \n\
fluxbox 2>&1 | sed "s/^/[fluxbox ] /" & \n\
pulseaudio --load="module-null-sink sink_name=null" 2>&1 | sed "s/^/[pulseaudio] /" & \n\
sleep 1 \n\
/usr/bin/vlc --no-qt-privacy-ask --alsa-audio-device=pulse --aout=pulse http://ice2.somafm.com/groovesalad-128-aac 2>&1 | sed "s/^/[vlc ] /"'\
> /home/runner/start.sh
EXPOSE 5900
ENTRYPOINT ["/bin/bash", "/home/runner/start.sh"]run with: docker build -f Dockerfile -t rfbproxy . && docker run -it --rm -p 5900:5900 rfbproxyto get some nice tunes courtesy of https://somafm.com/groovesalad/ on ws://localhost:5900/ |
|
Thanks. That image seems to work fine as constructed. I get a noticeable latency in the audio though, but that is somewhat expected given that this protocol has no synchronisation. However, only VLC seems to work properly. If I use mplayer then the stuttering is back. Can you see if it's the same when you test it? Are things perhaps reliant on the application providing a certain amount of buffered data? |
|
@lhchavez: Any news about it? |
|
Hey! I'm from Replit and would love to land this instead of maintaining our own fork. What would you like to see for this PR to land? For what it's worth, we have been using this in production for a few months now and people seem to be happy with the audio. |
|
Please see my previous comment. I could not get this to work reliably, which means I don't feel confident it will work reliably for our users. Have you applied any more fixes in your fork that aren't included here? |
|
No response for two years. Closing. |
6 participants
This change builds upon #1457 to support encoded audio support for the
RFB protocol. Notably, it:
a) Adds a button to toggle the audio stream. This is needed since most
browsers require the audio play action to be run in a user event
handler, so having the toggle audio be an explicit button achieves
that (and is also to avoid annoying users).
b) Drops support for the QEMU audio extension and instead uses the
Replit Audio messages, which allows for explicit negotiation for
audio compression.
c) Makes the audio library more robust and now works in Chrome and
Firefox.
The protocol extension and a first implementation can be found at https://github.com/replit/rfbproxy (tested with TigerVNC as the underlying server).
Tested in Chrome and Firefox (missing a volunteer to fix in Safari. maybe @tinyzimmer can help?)